Multimodal End-to-End Autonomous Driving

A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) able to drive towards a desired destination. Today, there are different paradigms addressing development AI drivers. On one hand, we find modular pipelines, which divide driving task into sub-tasks such as perception and maneuver planning control. other end-to-end approaches that try learn direct mapping from input raw sensor data control signals. The later relatively less studied, but gaining popularity since they demanding in terms annotation. This paper focuses on driving. So far, most proposals relying this paradigm assume RGB images data. However, AVs will not be equipped only with cameras, also active sensors providing accurate depth information ( e.g. , LiDARs). Accordingly, analyses whether combining modalities, xmlns:xlink="http://www.w3.org/1999/xlink">i.e. using RGBD data, produces better drivers than single modality. We consider multimodality based early, mid late fusion schemes, both multisensory single-sensor (monocular estimation) settings. Using CARLA simulator conditional imitation learning (CIL), show how, indeed, early outperforms single-modality.